Papers with semantic parsing evaluations
Calibrated Interpretation: Confidence Estimation in Semantic Parsing (2023.tacl-1)
Copied to clipboard
| Challenge: | Sequence generation models are increasingly being used to translate natural language into programs . calibration of such models is a key component of safety, says aaron sagar . |
| Approach: | They investigate whether calibration of popular generation models varies across models and datasets . they find that calibration varies among models and data sets, and that it is important to include it in evaluations if it is included . |
| Outcome: | The calibration of popular generation models varies across models and datasets . the authors find that the accuracy of models is dependent on confidence . |
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022.emnlp-main)
Copied to clipboard
Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, Kristina Toutanova
| Challenge: | Pre-trained language models struggle on out-of-distribution compositional generalization . recent work shows considerable improvements on many NLP tasks from model scaling . |
| Approach: | They evaluate encoder-decoder models up to 11B parameters and decoder-only models up 540B parameters . they compare scaling curves for fine-tuning, prompt tuning, and in-context learning methods . |
| Outcome: | The proposed scaling methods improve compositional generalization on many tasks . fine-tuning generally has flat or negative scaling curves on out-of-distribution compositional . larger models are better at modeling the syntax of the output space, the study finds . |